feat: Create Spark 4 Client #3026
Conversation
snazy
left a comment
snazy
left a comment
There was a problem hiding this comment.
Haven't reviewed the change in-depth, but I realize that this adds another quarkus-build to the project. Quarkus builds are not cheap (think: quick).
It would be much more efficient for both local builds/checks and CI duration to remove the quarkus builds from these plugins entirely.
This PR is meant to allow such usages by providing the URLs to a running Polaris/Quarkus instance to the tests, meaning that those tests work against a "real" Polaris instance.
|
@snazy - Good point about yet another Quarkus build! Given that this is an incremental move forward to support Spark 4. I would recommend that we merge this and, then, move to the Apprunner approach. Is that OK or do you want me to wait for the Apprunner? The reason that I ask is it'll allow us to be incrementally better and adjust as we go. |
github-project-automation
bot
moved this from PRs In Progress
to Done
in Basic Kanban Board
2 participants
Context
This change fixes #3021 by creating the Spark 4 client. Most of the changes are just simply copying and pasting what already exists.
I will carve out the overlapping business logic into a separate module later.
Test
Move over the same testing framework with what exists in the 3.5 plugin. In addition, I tested E2E with Parquet.
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed). I need the updates in test: Add Some Spark Client Tests and Update Documentation on Generic Tables #3000 first.